Parallelism config + TP + HSDP + BYODM (Bring Your Own Device Mesh)#3682
Merged
S1ro1 merged 76 commits intoJul 30, 2025
Merged
Conversation
salmanmohammadi
commented
Jul 16, 2025
salmanmohammadi
commented
Jul 16, 2025
salmanmohammadi
commented
Jul 16, 2025
10 tasks
added 2 commits
July 21, 2025 12:13
salmanmohammadi
changed the title
winglian
reviewed
Jul 21, 2025
winglian
reviewed
Jul 21, 2025
winglian
force-pushed
the
device_mesh_parallelism_config
branch
from
d9aec5c to
a402faf
Compare
S1ro1
reviewed
Jul 28, 2025
SunMarc
reviewed
Jul 29, 2025
Comment on lines
2775
to
2777
| def _prepare_fsdp2(self, *args): | ||
| # First pass: prepare everything except schedulers (and model, which is prepared separately below) | ||
| result = [ |
Member
There was a problem hiding this comment.
nice ! We will be able to clean up a bit trainer code after that
| def build_device_mesh(self, device_type: str): | ||
| mesh = self.get_mesh() | ||
| if not len(list(mesh)): | ||
| return |
Member
There was a problem hiding this comment.
yeah we should probably raise an error but tbh we don't really need to deal with this case
|
|
||
| ### FSDP2 + ND Parallelism | ||
|
|
||
| With `AccelerateDistributedConfig`, you can use 🤗 accelerate to train with n-dimensional parallelism. Script `nd_parallel.py` showcases just how you can do it. We enable you to configure 3 different parallel dimensions: |
Member
There was a problem hiding this comment.
You mean ParallelConfig no ?
Suggested change
| With `AccelerateDistributedConfig`, you can use 🤗 accelerate to train with n-dimensional parallelism. Script `nd_parallel.py` showcases just how you can do it. We enable you to configure 3 different parallel dimensions: | |
| With `ParallelConfig`, you can use 🤗 accelerate to train with n-dimensional parallelism. Script `nd_parallel.py` showcases just how you can do it. We enable you to configure 3 different parallel dimensions: |
Contributor
Author
There was a problem hiding this comment.
I think we have both because of the duplicate config upstream in transformers - but it would be good to clarify which to use.
Member
There was a problem hiding this comment.
It will be better to use the one from accelerate
SunMarc
reviewed
Jul 29, 2025
SunMarc
approved these changes
Jul 30, 2025
| <Tip> | ||
| Only use TP intra-node - therefore max TP size you should need is 8, you can also lower this as FSDP (`--dp-shard-size`) can be faster on smaller models with | ||
| shorter sequence lengths. If you still cannot fit into memory, utilize `--dp-shard-size` as much as you can. Then to scale up to utilize all your GPUs, fill the rest | ||
| with `--dp-replicate-size`. This is only a general guideline, you can (and should) experiment with different parallelism configurations to find the best one for your model and hardware. You can learn more about the general strategies for parallelism in our [blog](TODO) or if you wanna dive deep, read the [Ultra-Scale Playbook](https://huggingface.co/spaces/nanotron/ultrascale-playbook). |
Contributor
There was a problem hiding this comment.
Well the blog isn't ready, so we kind of need to keep the todo there haha (we'll finish before release)
Comment on lines
+752
to
+753
| def parallelism_config(self) -> ParallelismConfig | None: | ||
| return self.state.parallelism_config |
Member
There was a problem hiding this comment.
| syntax only works with py3.10 but we still need to support py3.9
Member
There was a problem hiding this comment.
we will drop py3.9 in october btw !
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Building on #3651
Dependencies:
ParallelismConfigin HFTrainerFixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
cc @S1ro1
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.